Image processing device, image processing method and program

ABSTRACT

An image processing device may include a control unit. The control unit may be operable, in a first mode, to control display on a display screen of a representative image of a group of images and, in a second mode, to control display on the display screen of an image of the group of images corresponding to a posture of the device. In addition, the control unit may be operable to switch between operation in the first mode and the second mode.

CROSS-REFERENCE TO RELATED APPLICATION

The present application is a national phase entry under 35 U.S.C. §371of International Application No. PCT/JP2011/000616 filed Feb. 3, 2011,published on Sep. 9, 2011 as WO 2011/108190 A1, which claims priorityfrom Japanese Patent Application No. JP 2010-048751 filed in theJapanese Patent Office on March 5, 2010.

TECHNICAL FIELD

The present invention relates to an image processing device, and moreparticularly, to an image processing device and an image processingmethod for displaying an image, and a program for causing a computer toexecute the method.

BACKGROUND ART

In recent years, imaging devices such as digital still cameras ordigital video cameras (e.g., camera-integrated recorders) for imaging asubject such as a person or an animal to generate image data andrecording the image data as image content have been used. Also, animaging device capable of determining a plurality of thus recorded imagecontents as one group according to a user's preference and performingcontent management (e.g., management in a photographing date unit) inthe determined group unit has been proposed. For example, the contentmanagement can be performed by determining a plurality of image contentsassociated with one another as one group.

A number of reproduction apparatuses for displaying the thus recordedimage contents have been proposed. For example, there is a reproductionapparatus for performing image advancing by user manipulation using amanipulation member and sequentially displaying a plurality of images.Also, there is a reproduction apparatus in which a user changes aposture of the reproduction apparatus to change displayed content of adisplay unit.

For example, an information processing device for obtaining a movementamount or a rotation amount of a body and instructing, for example, toscroll displayed content of a display unit according to the amount hasbeen proposed (e.g., see Patent Literature 1).

CITATION LIST Patent Literature

[PTL 1]

Japanese Patent Application Laid-Open No. H06-4208 (FIG. 3)

SUMMARY OF INVENTION Technical Problem

According to the above-described related art, the displayed content ofthe display unit can be changed by changing the posture of a device,making it possible for a user to easily perform a changing manipulationin a state in which the user holds the device by hand.

Here, in the case in which desired image content is displayed when thecontent management is performed in the group unit, for example,selection of a group to which the desired image content belongs andsequential display of image contents belonging to the group is assumedto be performed. However, image display switching and the change of theposture of the device for performing the display switching are assumednot to correspond to each other according to the type of image contentbelonging to each group. In this case, when a plurality of imagesassociated with one another belonging to the same group is displayed,each image is assumed not to be easily viewed.

In the light of the foregoing, it is desirable to display each image tobe easily viewed when a plurality of images associated with one anotherare displayed.

Solution to Problem

In accordance with one aspect of the invention, an image processingdevice may include a control unit operable, in a first mode, to controldisplay on a display screen of a representative image of a group ofimages and, in a second mode, to control display on the display screenof an image of the group of images corresponding to a posture of thedevice. In addition, the control unit may be operable to switch betweenoperation in the first mode and the second mode.

In accordance with another aspect of the invention, a method of imageprocessing may include, in a first mode, controlling display on adisplay screen of a representative image of a group of images, and, in asecond mode, controlling display on the display screen of an image ofthe group of images corresponding to a posture of the device. The methodfurther may include switching between the first mode and the secondmode.

In accordance with another aspect of the invention, a non-transitoryrecording medium may be recorded with a computer-readable programexecutable by a computer for enabling the computer to perform imageprocessing. The program may include the steps of, in a first mode,controlling display on a display screen of a representative image of agroup of images, and, in a second mode, controlling display on thedisplay screen of an image of the group of images corresponding to aposture of the device. The program may further include the step ofswitching between the first mode and the second mode.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1A shows an appearance configuration example of an imaging device100 in a first embodiment of the present invention.

FIG. 1B shows an appearance configuration example of an imaging device100 in a first embodiment of the present invention.

FIG. 2A shows one example of posture when the imaging device 100 is usedin the first embodiment of the present invention.

FIG. 2B shows one example of posture when the imaging device 100 is usedin the first embodiment of the present invention.

FIG. 2C shows one example of posture when the imaging device 100 is usedin the first embodiment of the present invention.

FIG. 3 is a block diagram showing a functional configuration example ofthe imaging device 100 in the first embodiment of the present invention.

FIG. 4 schematically shows stored content of the image managementinformation storage unit 210 in the first embodiment of the presentinvention.

FIG. 5 schematically shows an image display transition example on theinput/output panel 101 in the first embodiment of the present invention.

FIG. 6 schematically shows an image display transition example on theinput/output panel 101 in the first embodiment of the present invention.

FIG. 7A shows a relationship between images displayed on theinput/output panel 101 and postures of the imaging device 100 in thefirst embodiment of the present invention.

FIG. 7B shows a relationship between images displayed on theinput/output panel 101 and postures of the imaging device 100 in thefirst embodiment of the present invention.

FIG. 7C shows a relationship between images displayed on theinput/output panel 101 and postures of the imaging device 100 in thefirst embodiment of the present invention.

FIG. 8A schematically shows a relationship between a display transitionand the posture of the imaging device 100 in the input/output panel 101of the first embodiment of the present invention.

FIG. 8B schematically shows a relationship between a display transitionand the posture of the imaging device 100 in the input/output panel 101of the first embodiment of the present invention.

FIG. 9A schematically shows a relationship between a display transitionand the posture of the imaging device 100 in the input/output panel 101of the first embodiment of the present invention.

FIG. 9B schematically shows a relationship between a display transitionand the posture of the imaging device 100 in the input/output panel 101of the first embodiment of the present invention.

FIG. 10A shows a display transition example in the input/output panel101 in the first embodiment of the present invention.

FIG. 10B shows a display transition example in the input/output panel101 in the first embodiment of the present invention.

FIG. 11A schematically shows a relationship between a display transitionexample in the input/output panel 101 and the posture of the imagingdevice 100 in the first embodiment of the present invention.

FIG. 11B schematically shows a relationship between a display transitionexample in the input/output panel 101 and the posture of the imagingdevice 100 in the first embodiment of the present invention.

FIG. 11C schematically shows a relationship between a display transitionexample in the input/output panel 101 and the posture of the imagingdevice 100 in the first embodiment of the present invention.

FIG. 11D schematically shows a relationship between a display transitionexample in the input/output panel 101 and the posture of the imagingdevice 100 in the first embodiment of the present invention.

FIG. 12A schematically shows a relationship between a display transitionexample in the input/output panel 101 and the posture of the imagingdevice 100 in the first embodiment of the present invention.

FIG. 12B schematically shows a relationship between a display transitionexample in the input/output panel 101 and the posture of the imagingdevice 100 in the first embodiment of the present invention.

FIG. 12C schematically shows a relationship between a display transitionexample in the input/output panel 101 and the posture of the imagingdevice 100 in the first embodiment of the present invention.

FIG. 13A schematically shows a relationship between a display transitionexample in the input/output panel 101 and the posture of the imagingdevice 100 in the first embodiment of the present invention.

FIG. 13B schematically shows a relationship between a display transitionexample in the input/output panel 101 and the posture of the imagingdevice 100 in the first embodiment of the present invention.

FIG. 13C schematically shows a relationship between a display transitionexample in the input/output panel 101 and the posture of the imagingdevice 100 in the first embodiment of the present invention.

FIG. 14 is a flowchart showing an example of a processing procedure ofan image content reproduction process in the imaging device 100according to the first embodiment of the present invention.

FIG. 15 is a flowchart showing an example of a processing procedure ofan image content reproduction process in the imaging device 100according to the first embodiment of the present invention.

FIG. 16 schematically shows a flow of an interleaving process in thegroup image reproduction unit 160 according to the second embodiment ofthe present invention.

FIG. 17 is a flowchart showing an example of a processing procedure ofan image content reproduction process in the imaging device 100according to the second embodiment of the present invention.

DESCRIPTION OF EMBODIMENTS

Hereinafter, preferred embodiments of the present invention will bedescribed in detail with reference to the appended drawings. Note that,in this specification and the appended drawings, structural elementsthat have substantially the same function and structure are denoted withthe same reference numerals, and repeated explanation of thesestructural elements is omitted.

The description will be given in the following order.

-   1. First embodiment (display control: an example in which images    belonging to a group corresponding to a representative image    designated in a representative image display mode are displayed in a    group image display mode)-   2. Second embodiment (display control: an example in which images to    be displayed are subjected to an interleaving process in a group    image display mode)    1. First Embodiment    (Appearance Configuration Example of Imaging Device)

FIGS. 1A and 1B show an appearance configuration example of an imagingdevice 100 in a first embodiment of the present invention. FIG. 1A is aperspective view showing an appearance at a display surface side of theimaging device 100. FIG. 1B is a side view showing an appearance at thedisplay surface side of the imaging device 100. The imaging device 100is realized, for example, by a digital still camera or a digital videocamera (e.g., camera-integrated recorder) capable of imaging a subjectto generate image data, recording the image data as image content, anddisplaying the image content.

The imaging device 100 includes an input/output panel 101 and a shutterbutton 102. The imaging device 100 is an image processing device thatcan be carried by a user and the user can view an image displayed on theinput/output panel 101. The imaging device 100 is one example of animage processing device defined in the claims.

The input/output panel 101 displays various images and receives amanipulation input from a user by detecting a contact manipulation onthe input/output panel 101. The shutter button 102 is a button pressedby a photographer when image data (an imaged image) generated by imaginga subject is recorded as image content (a still image file).

The imaging device 100 includes other manipulation members such as apower switch and a mode switching switch, a lens unit, and the like, butsuch manipulation members will not be shown and described herein forease of explanation. Here, the lens unit (not shown) is an opticalsystem including a plurality of lenses for focusing light from thesubject, an aperture, and the like.

Here, a change of posture of the imaging device 100 will be described.For example, the user may change rotation angles (i.e., a yaw angle, apitch angle, and a roll angle) about three orthogonal axes in a state inwhich the user holds the imaging device 100 by hand. For example, theuser may change the posture of the imaging device 100 in a directionindicated by the arrow 302 (the yaw angle), the axis of which is thearrow 300 (or the arrow 301). The change example is shown in FIG. 2B.For another example, the user may change the posture of the imagingdevice 100 in a direction indicated by the arrow 305 (the pitch angle),the axis of which is the arrow 303 (or the arrow 304). For yet anotherexample, the user may change the posture of the imaging device 100 in adirection indicated by the arrow 308 (the roll angle), the axis of whichis the arrow 306 (or the arrow 307).

Further, for example, the user may change the posture of the imagingdevice 100 by moving (sliding) the imaging device 100 along a straightline on a plane in a state in which the user holds the imaging device100 by hand. For example, the user may change the posture by moving theimaging device 100 in the direction indicated by the arrow 300 or thearrow 301 (movement in a vertical direction in FIG. 1B). For anotherexample, the user may change the posture by moving the imaging device100 in a direction indicated by the arrow 303 or 304 (movement in ahorizontal direction in FIG. 1B). This change example is shown in FIG.2C. Further, for example, the user may change the posture by moving theimaging device 100 in a direction indicated by the arrow 306 or 307.

(Use Example of Imaging Device)

FIGS. 2A to 2C show one example of the posture when the imaging device100 is used in the first embodiment of the present invention. In FIG.2A, one example of the posture of the imaging device 100 when imagecontent reproduction is performed using the imaging device 100 is shown.When a person 310 performs the image content reproduction using theimaging device 100, the person 310 may view an image displayed on theinput/output panel 101, for example, in a state in which the person 310holds the imaging device 100 with both hands as shown in FIG. 2A.

In FIGS. 2B and 2C, a transition example in which the posture of theimaging device 100 is changed is shown. In FIG. 2B, a transition examplewhen the imaging device 100 is rotated in the direction indicated by thearrow 302 about the arrow 300 (301) shown in FIG. 1A to change theposture is shown.

In FIG. 2C, a transition example in which the imaging device 100 ismoved in the direction indicated by the arrows 303 and 304 shown inFIGS. 1A and 1B to change the posture is shown. In the first embodimentof the present invention, an example in which an image displayed on theinput/output panel 101 is sequentially changed by changing the postureof the imaging device 100 as such is shown. That is, an example in whichthe image displayed on the input/output panel 101 is sequentiallychanged by gesture manipulation by the user is shown. Such changes ofthe image display status will be described in detail with reference toFIGS. 7A to 13C.

(Functional Configuration Example of Imaging Device)

FIG. 3 is a block diagram showing a functional configuration example ofthe imaging device 100 in the first embodiment of the present invention.The imaging device 100 includes an input/output unit 110, a posturedetection unit 120, an analysis unit 130, a control unit 140, arepresentative image reproduction unit 150, a group image reproductionunit 160, a rendering memory 170, an image content storage unit 200, andan image management information storage unit 210.

The image content storage unit 200 stores image data (an imaged image)generated by the imaging unit (not shown) as an image file (imagecontent (still image content or moving image content)). The imagecontent storage unit 200 supplies the stored image content to therepresentative image reproduction unit 150 or the group imagereproduction unit 160. In the first embodiment of the present invention,an example in which the still image content is used as the image contentis shown.

The image management information storage unit 210 stores managementinformation (image management information) on the image content storedin the image content storage unit 200. Using the image managementinformation, reproduction in the representative image display mode andthe group image display mode is performed. Here, the representativeimage display mode is a mode in which a representative image of groupedimage contents and an image of non-grouped image content among the imagecontents stored in the image content storage unit 200 are sequentiallydisplayed according to user manipulation. The group image display modeis a mode in which images of each grouped image content among the imagecontents stored in the image content storage unit 200 are sequentiallydisplayed according to the change of the posture of the imaging device100. The image content storage unit 200 and the image managementinformation storage unit 210 may be, for example, one or a plurality ofremovable recording media, such as discs, such as digital versatilediscs (DVDs) or semiconductor memories such as memory cards. Therecording media may be embedded in the imaging device 100 or detachablyprovided in the imaging device 100.

The input/output unit 110 includes a display unit 111 and a manipulationreceiving unit 112. The display unit 111 is a display unit fordisplaying an image supplied from the representative image reproductionunit 150 or the group image reproduction unit 160. Various menu screensor various images are displayed on the display unit 111. The displayunit 111 may be, for example, a liquid crystal display (LCD) or anorganic electro luminescence (EL) panel.

The manipulation receiving unit 112 is a manipulation receiving unit forreceiving content of a manipulation by the user, and supplies amanipulation signal dependent on the received manipulation content tothe control unit 140. The manipulation receiving unit 112 correspondsto, for example, the manipulation member such as the shutter button 102shown in FIGS. 1A and 1B or a touch panel. For example, the touch panelis superimposed on the display unit 111 to transmit the screen of thedisplay unit 111, and receives a manipulation input from the user bydetecting an object contacting the display surface. That is, when a userpresses a desired location, for example, by touching the display surfacein the input/output unit 110 with his or her finger, the touch paneldetects a coordinate of the pressed location and outputs a manipulationsignal corresponding to the detected coordinate to the control unit 140.When the control unit 140 acquires the manipulation signal, the controlunit 140 executes a predetermined process based on the acquiredmanipulation signal.

The manipulation receiving unit 112 may be, for example, a touch panelfor converting a contact on a display surface into an electrical signalusing a dedicated sensing device and outputting the converted electricalsignal to the control unit 140. Further, the manipulation receiving unit112 may be, for example, a touch panel of an optical sensor type capableof detecting an object (e.g., a user's finger) approaching or contactinga display surface. The input/output unit 110 corresponds to amanipulation member such as the shutter button 102 or the input/outputpanel 101 shown in FIGS. 1A and 1B.

The posture detection unit 120 detects a change of the posture of theimaging device 100 by detecting acceleration, movement, tilt and thelike of the imaging device 100, and outputs posture change informationabout the detected change of the posture to the analysis unit 130. Forexample, the posture detection unit 120 detects a movement direction anda movement amount in a specific direction of the imaging device 100 asthe change of the posture of the imaging device 100. The posturedetection unit 120 may be realized by a gyro sensor (angular velocitysensor) or an acceleration sensor. The posture detection unit 120 is anexample of a detection unit defined in the claims.

The analysis unit 130 analyzes the changed amount of the posture of theimaging device 100 (e.g., the movement direction and the movementamount) based on the posture change information output from the posturedetection unit 120 and determines whether the image displayed on thedisplay unit 11 should be switched (image advancing or image returning).The analysis unit 130 outputs the determination result (an image displayswitching instruction) to the group image reproduction unit 160.Specifically, when the group image display mode is set, the control unit140 outputs an analysis instruction and analysis reference information(determination reference information (e.g., a manipulation method 215shown in FIG. 4) used for the determination as to whether the imageshould be switched) to the analysis unit 130. Further, a location of theposture of the imaging device 100 when the analysis instruction isoutput is used as a reference location, and according to the analysisreference information, the analysis unit 130 determines whether theimage should be switched based on the posture change information outputfrom the posture detection unit 120. The analysis unit 130 outputs thedetermination result to the group image reproduction unit 160. Forexample, the analysis unit 130 determines whether the posture of theimaging device 100 is changed above a certain amount based on theposture change information output from the posture detection unit 120.Subsequently, when the posture of the imaging device 100 is changedabove a certain amount, the analysis unit 130 determines whether thechange corresponds to a manipulation method (the manipulation method 215shown in FIG. 4) according to the analysis reference information.Subsequently, when the change corresponds to the manipulation method,the analysis unit 130 outputs the image display switching instruction(an image advancing or image returning instruction) to the group imagereproduction unit 160 based on the change amount. The determination asto whether image should be switched will be described in detail withreference to FIGS. 7A to 7C.

The control unit 140 controls each unit of the imaging device 100 basedon the manipulation content from the manipulation receiving unit 112.For example, the control unit 140 sets the representative image displaymode when the manipulation receiving unit 112 receives a representativeimage display mode setting manipulation and sets the group image displaymode when the manipulation receiving unit 112 receives a group imagedisplay mode setting manipulation.

Further, when the group image display mode setting manipulation isreceived, the control unit 140 controls to sequentially display, on thedisplay unit 111, the images in the group corresponding to therepresentative image displayed on the display unit 11 upon a settingmanipulation. When the group image display mode is set, the control unit140 controls to sequentially display, on the display unit 111, theimages in the group, based on the posture change detected by the posturedetection unit 120 and the correlativity between the images in thegroup. Here, the correlativity between the images in the group refersto, for example, correlativity between the images upon imaging by theimaging device 100. For example, an indication that a plurality ofimages generated by changing a view for a target subject (e.g., a faceof a person) as shown in FIG. 7A are multi-view images is obtained asthe correlativity. In this case, as shown in FIG. 7B, the images in thegroup can be sequentially displayed by horizontally rotating the imagingdevice 100. For example, an indication that a plurality of imagesgenerated by moving the imaging device 100 according to a movement ofthe target subject (e.g., person) as shown in FIG. 11D are operationstate images of the moving target subject is obtained as thecorrelativity. In this case, the images in the group can be sequentiallydisplayed by horizontally rotating the imaging device 100 as shown inFIG. 11 C. Further, for example, an indication that a plurality ofimages produced by imaging a moving target subject (e.g., airplane)using a fixed imaging device 100 as shown in FIG. 12C are horizontalmovement images is obtained as the correlativity. In this case, theimages in the group can be sequentially displayed by horizontally movingthe imaging device 100, as shown in FIG. 12A. The manipulation method215 shown in FIG. 4 is obtained based on the correlativity.

When the representative image display mode has been set and aninstruction manipulation to instruct display switching (e.g., imageadvancing and image returning) of the image displayed on the displayunit 11 is received, the control unit 140 instructs the representativeimage reproduction unit 150 to switch the image display based on theinstruction manipulation. Further, when a group image display modesetting manipulation is performed, the control unit 140 acquires amanipulation method (the manipulation method 215 shown in FIG. 4)associated with the group corresponding to the representative imagedisplayed on the display unit 11 upon a setting manipulation from thegroup image reproduction unit 160. The control unit 140 outputs theanalysis reference information and the analysis instruction to theanalysis unit 130 according to the acquired manipulation method. Thecontrol unit 140 is realized by, for example, a central processing unit(CPU).

The representative image reproduction unit 150 displays, on the displayunit 111, image content corresponding to the representative image andnon-grouped image content among the image contents stored in the imagecontent storage unit 200 under control of the control unit 140.Specifically, when the representative image display mode has been set,the representative image reproduction unit 150 acquires the imagecontent corresponding to the representative image and non-grouped imagecontent from the image content storage unit 200. Subsequently, therepresentative image reproduction unit 150 decodes the acquired imagecontent and renders a display image in the rendering memory 170 based onthe decoded image content. The representative image reproduction unit150 sequentially displays one of the images rendered in the renderingmemory 170 on the display unit 111 under control of the control unit140. The representative image reproduction unit 150 displays, on thedisplay unit 111, the representative image, and manipulation supportinformation (e.g., the manipulation support information 401 shown inFIG. 6( a)) supporting the setting manipulation to the group imagedisplay mode for displaying each image in the group corresponding to therepresentative image.

The group image reproduction unit 160 displays the grouped imagecontents among the image contents stored in the image content storageunit 200 on the display unit 111 in a group unit under control of thecontrol unit 140. Specifically, when the group image display mode isset, the group image reproduction unit 160 acquires each image contentbelonging to the group corresponding to the representative imagedisplayed upon the group image display mode setting manipulation fromthe image content storage unit 200. Subsequently, the group imagereproduction unit 160 decodes the acquired image content and renders adisplay image in the rendering memory 170 based on the decoded imagecontent. In this case, the group image reproduction unit 160 arrangesdisplay images and renders the images in the rendering memory 170, forexample, according to a predetermined rule.

When the group image display mode setting manipulation is received, thegroup image reproduction unit 160 displays manipulation supportinformation indicating a manipulation method associated with a group tobe displayed, and the respective images in the group on the display unit111 to be associated with each other. In this case, for example, thegroup image reproduction unit 160 changes content of the manipulationsupport information based on the change of the posture of the imagingdevice 100 detected by the posture detection unit 120. That is, thegroup image reproduction unit 160 sequentially displays, on the displayunit 111, one image from among the images rendered in the renderingmemory 170 based on the image display switching instruction output fromthe analysis unit 130 (an image advancing or returning instruction).

For example, the group image reproduction unit 160 determines whetherthe manipulation method corresponding to the manipulation supportinformation is performed based on the change of the posture, and whenthe manipulation method is performed, the group image reproduction unit160 may delete the manipulation support information.

The rendering memory 170 is a rendering buffer for holding the imagesrendered by the representative image reproduction unit 150 or the groupimage reproduction unit 160, and supplies the rendered images to therepresentative image reproduction unit 150 or the group imagereproduction unit 160.

(Stored Content of Image Management Information Storage Unit)

FIG. 4 schematically shows stored content of the image managementinformation storage unit 210 in the first embodiment of the presentinvention. Group identification information 211, image contentidentification information 212, group presence 213, a representativeimage 214, and a manipulation method 215 are stored in the imagemanagement information storage unit 210.

Identification information for identifying each group is stored in thegroup identification information 211. Here, the group is an image groupincluding a plurality of image contents, associated with one another,the order of which is specified based on a predetermined rule. The groupincludes, for example, images 411 to 417 shown in FIGS. 7A and 8A. Inthe first embodiment of the present invention, an example in which animage group including a plurality of image contents, and image contentsnot in the image group are assigned group identification information toidentify the image contents not in the image group and the image groupis shown.

Identification information (image content identification information)for identifying each image content stored in the image content storageunit 200 is stored in the image content identification information 212.For example, the image content identification information of each imagecontent is stored in image advancing or returning order (e.g., order ofphotographing time (record date)).

Information for identifying whether the image contents are grouped(whether the image contents belong to a group) is stored in the grouppresence 213. In the example shown in FIG. 4, “present” is indicated ina field of the grouped image content and “absent” is indicated in afield of the non-grouped image content.

Representative image information for specifying a representative imageamong a plurality of grouped image contents is stored in therepresentative image 214. In the example shown in FIG. 4, image contentidentification information (an image content identification number) ofimage content corresponding to the representative image is shown as therepresentative image information. Further, “absent” is indicated in afield of the non-grouped image content.

A manipulation method when a plurality of grouped image contents issequentially displayed in the case in which the group image display modehas been set is stored in the manipulation method 215. The manipulationmethod is a manipulation method relating to the change of the posture ofthe imaging device 100. For example, “horizontal rotation” refers to amanipulation to sequentially display a plurality of image contents byrotating the imaging device 100 in the direction indicated by the arrow302 about the arrow 300 (301) shown in FIGS. 1A and 1B to change theposture. Further, for example, “vertical rotation” refers to amanipulation to sequentially display a plurality of image contents byrotating the imaging device 100 in the direction indicated by the arrow305 about the arrow 303 (304) shown in FIGS. 1A and 1B to change theposture. Further, for example, “horizontal movement” refers to amanipulation to sequentially display a plurality of image contents bymoving the imaging device 100 in the direction indicated by the arrows303 and 304 shown in FIGS. 1A and 1B to change the posture. For example,“vertical movement” refers to a manipulation to sequentially display aplurality of image contents by moving the imaging device 100 in thedirection indicated by the arrows 300 and 301 shown in FIGS. 1A and 1Bto change the posture. The imaging device 100 can automaticallydetermine the manipulation method, for example, according to an imagingoperation method upon generation of the image contents. Further, themanipulation method may be appropriately determined according to auser's preference by the user manipulation.

The information stored in the group identification information 211, thegroup presence 213, the representative image 214 and the manipulationmethod 215 may be recorded in an image file (image content). Forexample, such information can be recorded as attribute informationrelating to a still image file. The information recorded in the imagecontent may be used upon image content reproduction.

(Image Display Transition Example)

FIGS. 5 and 6 schematically show image display transition examples onthe input/output panel 101 in the first embodiment of the presentinvention. In FIG. 5( a), a case in which the image contents stored inthe image content storage unit 200 are arranged according to apredetermined rule (order of image content identification information)is schematically shown. In the example shown in FIG. 5( a), only imagecontents #1 to #13 among the image contents stored in the image contentstorage unit 200 are shown. Further, the image contents #1 to #13correspond to #1 to #13 indicated in the image content identificationinformation 212 of FIG. 4. In FIG. 5, image contents and correspondingimages are schematically shown by rectangles with image contentidentification information (#1 to #13). Further, grouped image contentsare surrounded in a group unit by dotted rectangles.

In FIG. 5( b), a transition of images displayed on the input/outputpanel 101 when the representative image display mode has been set isshown. As shown in FIG. 5( b), non-grouped image contents (#1, #5, #6,and #13) and representative images (#3 and #9) of the grouped imagecontents are sequentially displayed based on an image advancingmanipulation or an image returning manipulation by the user. The imageadvancing manipulation or the image returning manipulation is performed,for example, using the manipulation member included in the imagingdevice 100 (e.g., the touch panel or an external manipulation button(not shown)). Further, the image advancing manipulation or the imagereturning manipulation may be performed, for example, according to thechange of the posture of the imaging device 100, similar to setting ofthe group image display mode.

In FIG. 6( a), a display transition example in which image contents #6,#9 and #13 among the images shown in FIG. 5( b) are sequentiallydisplayed based on the image advancing manipulation or the imagereturning manipulation by the user is shown. In FIG. 6, imagesindicating image contents using rectangles with image contentidentification information (#6, #8 to #10, and #13) are schematicallyshown. Thus, it is possible to display each image on the input/outputpanel 101 based on the user manipulation. Here, when the representativeimage (e.g., #9) of the grouped image content is displayed on theinput/output panel 101 as shown in FIG. 6( a), manipulation supportinformation 401 for setting the group image display mode is displayed.The manipulation support information 401 is a manipulation button forsupporting a manipulation for setting the group image display mode uponsetting of the representative image display mode. That is, the user canset the group image display mode by pressing a rectangular areacorresponding to the manipulation support information 401. For example,the group image display mode is set when the manipulation supportinformation 401 is pressed in a state in which representative images(e.g., #3 and #9) of the grouped image contents are displayed on theinput/output panel 101. Thus, the manipulation to press the manipulationsupport information 401 becomes an entrance to the group image displaymode. Where the group image display mode is set, the images contentsbelonging to the group corresponding to the representative imagedisplayed when the group image display mode setting manipulation (amanipulation to press the manipulation support information 401) isperformed are sequentially displayed.

On the other hand, when the non-grouped image contents (e.g., #6 and#13) are displayed on the input/output panel 101 as shown in FIG. 6( a),the manipulation support information 401 is not displayed. Accordingly,the manipulation to press the manipulation support information 401 willnot be performed and the group image display mode will not be set in astate in which the non-grouped image contents (e.g., #6 and #13) havebeen displayed on the input/output panel 101.

Here, a case in which a group image display mode setting manipulation(the manipulation to press the manipulation support information 401) isperformed in a state in which the representative image (#9) of thegrouped image content has been displayed on the input/output panel 101is assumed. In this case, the group image reproduction unit 160 acquiresimage contents belonging to the group corresponding to therepresentative image displayed when the group image display mode settingmanipulation is performed, from the image content storage unit 200. Thegroup image reproduction unit 160 renders each image in the renderingmemory 170 based on the acquired image content. In this case, the groupimage reproduction unit 160 renders each image in the rendering memory170 according a predetermined rule (e.g., the order of image contentidentification information). An image rendering example in the renderingmemory 170 is shown in FIG. 5( c).

In FIG. 5( c), a case in which the image contents belonging to the groupcorresponding to the representative image displayed when the group imagedisplay mode is set are rendered in the rendering memory 170 isschematically shown. When an image content rendering process is thusperformed, the group image reproduction unit 160 displays the fact onthe input/output panel 101 and does not perform a displaying process inthe group image display mode. A display example in this case is shown inFIG. 6( b).

In FIG. 6( b), a display example when the image content renderingprocess is performed directly after the group image display mode settingmanipulation is performed is shown. For example, manipulation supportinformation 402 indicating the image content rendering processsuperimposed on the representative image (#9) displayed when the groupimage display mode is set is displayed on the input/output panel 101.When the image content rendering process is terminated, the group imagereproduction unit 160 displays manipulation support information forsupporting the image display switching manipulation in the group imagedisplay mode on the input/output panel 101. The display example is shownin FIG. 6( c).

In FIG. 6( c), a display example when the image content renderingprocess is terminated after the group image display mode settingmanipulation is performed is shown. For example, manipulation supportinformation 403 and 404 for supporting the image display switchingmanipulation in the group image display mode superimposed on therepresentative image (#9) displayed when the group image display mode isset is displayed on the input/output panel 101. The manipulation supportinformation 403 and 404 is manipulation guides for supporting amanipulation for displaying each image in the group displayed on theinput/output panel 101. The manipulation support information 403 and 404shown in FIG. 6( c) shows that image advancing or image returning can beperformed by the user performing a manipulation to rotate the imagingdevice 100 about the arrow 300 (301) shown in FIG. 1A in a state inwhich the user holds the imaging device 100 in his or her hand. In therespective drawings, the manipulation support information is shown to berelatively large for ease of explanation. Further, the image advancingmanipulation and the image returning manipulation upon setting of thegroup image display mode will be described in detail with reference toFIGS. 7A to 13C.

Here, as shown in FIG. 6( c), a setting manipulation to set therepresentative image display mode in a state in which the group imagedisplay mode is set and the manipulation support information 403 and 404superimposed on the image to be displayed is displayed will bedescribed. When the representative image display mode is set in thisstate, for example, the user performs a touch manipulation to contactthe display surface of the input/output panel 101. When the touchmanipulation is performed, the mode is switched from the group imagedisplay mode to the representative image display mode (i.e., therepresentative image display mode is set), as shown by the arrow 405.Directly after the representative image display mode is set as such, arepresentative image (e.g., image #9) of the group displayed uponsetting of the group image display mode is displayed. The representativeimage display mode may be set by a setting manipulation using themanipulation member, as well as the touch manipulation in theinput/output panel 101.

(Example of Relationship Between Change of Posture of Imaging Device andImage Display Transition Upon Setting of Group Image Display Mode)

FIGS. 7A to 7C show a relationship between images displayed on theinput/output panel 101 and postures of the imaging device 100 in thefirst embodiment of the present invention. In FIG. 7A, images 411 to 417continuously displayed when the group image display mode has been setare arranged according to a predetermined rule. The images 411 to 417belong to the same group, and the image 414 is a representative image.Further, the images 411 to 417 are sequentially displayed in the ordershown in FIG. 7A according to the change of the posture of the imagingdevice 100.

In FIG. 7B, a manipulation range of the imaging device 100 when theimages 411 to 417 are displayed is shown. In this example, an example inwhich the image is advanced or returned by rotating the imaging device100 about the arrow 300 (301) shown in FIGS. 1A and 1B to change theposture is shown. The rotation angle range in this case is (theta).

In FIG. 7C, an angle as a reference (reference angle) when imageadvancing or image returning is performed on the images 411 to 417 isshown. In this example, an example in which image advancing or imagereturning is performed on seven images (images 411 to 417) when therotation angle range (theta) is 60 degrees will be described. In thiscase, the reference angle (theta)1 when image advancing or imagereturning is performed may be 10 degrees (=60 degrees (rotation anglerange (theta))/6 (number of images 411 to 417)). Specifically, theanalysis unit 130 sets the location of the imaging device 100 as areference location 421 when the group image display mode is set. Whenthe rotation angle from the reference location 421 (rotation angle ofthe imaging device 100 when the arrow 300 (301) shown in FIGS. 1A and 1Bis used as an axis 420) exceeds 10 degrees, the analysis unit 130outputs an image display switching instruction to the group imagereproduction unit 160. When the image display switching instruction isoutput, the group image reproduction unit 160 performs image advancingor image returning based on content of the instruction.

For example, the representative image (image 414) is displayed on theinput/output panel 101 directly after the group image display mode isset. The user changes the posture of the imaging device 100 so that theleft hand side goes down in a state in which the person 310 holds theimaging device 100 as shown in FIG. 2A. In this case, if the rotationangle from the reference location 421 exceeds 10 degrees (when therotation angle exceeds the rotation angle range 422) when the groupimage display mode is set, image advancing is performed. That is, theimage 415 is displayed instead of the image 414. Similarly, each timethe rotation angle exceeds 10 degrees (each time the rotation angleexceeds the rotation angle ranges 423 and 424), image advancing isperformed and the images 416 and 417 are sequentially displayed.

On the other hand, the person 310 changes the posture of the imagingdevice 100 so that the right hand side goes down in the state in whichthe person 310 holds the imaging device 100 as shown in FIG. 2A. In thiscase, if the rotation angle from the reference location 421 exceeds 10degrees (exceeds the rotation angle range 425) when the group imagedisplay mode is set, image return is performed. That is, the image 413is displayed instead of the image 414. Similarly, each time the rotationangle exceeds 10 degrees (each time the rotation angle exceeds therotation angle ranges 426 and 427), the image returning is performed andthe images 412 and 411 are sequentially displayed.

FIGS. 8A to 9B schematically show a relationship between a displaytransition and the posture of the imaging device 100 in the input/outputpanel 101 of the first embodiment of the present invention. In FIG. 8A,a transition of an image displayed on the input/output panel 101 isshown. The images 411 to 417 displayed on the input/output panel 101shown in FIG. 8A are the same as the images 411 to 417 shown in FIG. 7A.In FIG. 8A, manipulation support information 411 to 417 displayed to besuperimposed on the image displayed on the input/output panel 101 isomitted. A display example of the manipulation support information isshown in FIG. 9A.

In FIG. 8B, one example of the posture of the imaging device 100 when animage is displayed on the input/output panel 101 shown in FIG. 8A isshown. In the example shown in FIG. 8B, a transition example in whichthe posture of the imaging device 100 upon setting of the group imagedisplay mode becomes parallel with a horizontal plane 330 is shown. InFIGS. 8A and 8B, the images displayed on the input/output panel 101shown in FIG. 8A and the postures of the imaging device 100 shown inFIG. 8B are arranged at the left and right to show a correspondencerelationship therebetween.

FIG. 9A shows a transition of an image displayed on the input/outputpanel 101. Images 411, 414 and 417 displayed on the input/output panel101 shown in FIG. 9A are the same as the images 411, 414 and 417 shownin FIG. 8A. In FIG. 9A, the images 412, 413, 415 and 416 displayed onthe input/output panel 101 are omitted. As described above, arepresentative image (image 414) is displayed on the input/output panel101 directly after the group image display mode is set. In this case,manipulation support information 403 and 404 superimposed on the image414 is displayed, as in FIG. 6( c). When the images 412, 413, 415 and416 are displayed on the input/output panel 101, the manipulationsupport information 403 and 404 superimposed on each image is similarlydisplayed.

If the image 411 has been displayed on the input/output panel 101 whenthe group image display mode has been set, further image returning willnot be performed. Accordingly, manipulation support information 431 and432 for supporting an image advancing manipulation superimposed on theimage 411 is displayed on the input/output panel 101, as shown in FIG.9A. Further, if the image 417 has been displayed on the input/outputpanel 101 when the group image display mode has been set, further imageadvancing will not be performed. Accordingly, manipulation supportinformation 433 and 434 for supporting image returning manipulationsuperimposed on the image 411 is displayed on the input/output panel101, as shown in FIG. 9A.

The manipulation support information may be deleted or changed, forexample, reduced, when image advancing and image returning are performedfor a certain time (or a certain number of times) by the usermanipulation.

In FIG. 9B, one example of the posture of the imaging device 100 whenthe image is displayed in the input/output panel 101 shown in FIG. 9A isshown. In FIG. 9B, some of the postures of the imaging device 100 shownin FIG. 8B are omitted, similar to FIG. 9A. In FIG. 9, the imagesdisplayed on the input/output panel 101 shown in FIG. 9A and thepostures of the imaging device 100 shown in FIG. 9B are arranged at theleft and right to show a correspondence relationship therebetween,similar to FIG. 8.

Thus, image advancing or image return of the image displayed on theinput/output panel 101 is performed by tilting the imaging device 100 tothe left and right. Further, image advancing or image returning can berapidly performed by the user rapidly performing a manipulation to tiltthe imaging device 100 to the left and right.

Here, the images 411 to 417 are images obtained by imaging a face of thesame person from different views. Accordingly, the user can obtain thefeeling of viewing a three dimensional object (a face of a person) byrapidly performing the user manipulation to tilt the imaging device 100to the left and right and rapidly performing image advancing or imagereturning. That is, the user manipulation to tilt the imaging device 100to the left and right enables the user to obtain the feeling of viewinga pseudo-three-dimensional image.

Further, in the example in which a pseudo-three-dimensional image isdisplayed by performing the user manipulation to tilt the imaging device100 to the left and right, other objects as well as the face of theperson may be subjects. Hereinafter, a display example in which an imageobtained by imaging the same object from different views is displayed isshown.

FIGS. 10A and 10B show a display transition example in the input/outputpanel 101 in the first embodiment of the present invention. In FIGS. 10Aand 10B, a display transition example in the input/output panel 101 whenthe same vehicle is an image to be displayed is shown. In FIGS. 10A and10B, some of the images and manipulation support information areomitted.

In FIGS. 10A and 10B, for example, a case in which the rotation anglerange (theta) (shown in FIG. 7B) is 180 degrees is shown by way ofexample. In this case, the user can three-dimensionally view the vehiclefrom left and right sides thereof to a front by performing amanipulation to tilt the imaging device 100 to the left and right (amanipulation to rotate the imaging device 100 about the arrow 300 (301)shown in FIGS. 1A and 1B).

Thus, it is possible to enjoy a pseudo-three-dimensional image as aprofound image by rapidly advancing an image group obtained byphotographing the target subject from several angles by a gesturemanipulation of the user. That is, when the user desires tothree-dimensionally view the object, such as a face of a person or avehicle, he or she can easily view a pseudo-three-dimensional image ofthe object through a manual manipulation.

The angle information (the rotation angle range (theta) and thereference angle (theta)1 shown in FIG. 7C) may be changed by the usermanipulation to be easily viewed according to user's preference.Further, the rotation angle range (the reference angle) used fordetermination upon image display switching may be changed for each imagebased on a relationship between adjacent or neighboring images.

Further, the angle information (the rotation angle range and thereference angle) is stored in the image management information storageunit 210 to be associated with each group, and a reproduction process inthe group image display mode may be performed using the stored angleinformation. In this case, the reference angles may be sequentiallyrecorded, for example, based on a location relationship between thetarget subject (e.g., a face of a person) and the imaging device 100 inthe imaging operation. For example, when the face of the person is thetarget subject, an imaging location from the front is 0 degrees andangles of the view from the imaging location 0 degrees using the face asa center (angles corresponding to the imaging location) are sequentiallyrecorded as the reference angles.

Further, the angle information (the rotation angle range and thereference angle) may be obtained by performing image processing on animage to be displayed. For example, a feature amount of each image isextracted and feature amounts between adjacent or neighboring images arecompared to calculate similarity between the feature amounts. Thereference angle may be determined based on the similarity. For example,when the similarity is high, an angle of two views between adjacentimages (an angle specified by two lines connecting the two views and thesubject (one point) (an angle with the point as a center)) is assumed tobe small, thus decreasing the reference angle. On the other hand, whenthe similarity is low, the angle of the two views between adjacentimages is assumed to be great, thus increasing the reference angle. Thefeature amount is data for calculating the similarity between images,and may be extracted, for example, based on a color or brightness value.For example, when a face of a person is included in the image, dataindicating a feature or a property of a face image for recognizing alocational relationship or shape of each portion such as eyes, nose,mouth or eyebrows constituting the face may be used as the featureamount. For example, a process of detecting a face included in eachimage (e.g., a face (person or animal) occupying the image above acertain percentage) may be performed, and when the face is detected, adirection of the face in each image may be detected, and the referenceangle may be determined based on the direction of the face. As a methodof detecting the face included in the image, for example, a facedetection method using matching between a template having brightnessdistribution information of the face recorded therein and a contentimage (e.g., see Japanese Patent Laid-open Publication No. 2004-133637)may be used. Alternatively, a face detection method based on a featureamount of a portion of skin color or a person's face included in theimage may be used. Furthermore, as a face direction detection method,for example, a detection method of detecting a face attribute (facedirection) by means of a weak classifier using a difference value inbrightness between two points in a face image may be used (e.g., seeJapanese Patent Laid-open Publication No. 2009-301170). In this case,the reference angle may be determined, for example, according to a sizeof a face direction determination score. Further, the rotation anglerange (theta) may be obtained based on each obtained reference angle(e.g., (theta)=sum of the reference angles). For example, the groupimage reproduction unit 160 performs calculation of such angleinformation under control of the control unit 140.

In the above description, the example in which a plurality of images issequentially displayed by performing a manipulation to tilt the imagingdevice 100 in a predetermined direction and is three-dimensionallyviewed has been shown. Here, a case in which a target subject includedin a plurality of images is desired to be dynamically viewed when theimages are sequentially displayed is also assumed. Further, when theplurality of images is sequentially displayed, a manipulation to movethe imaging device 100 in a predetermined direction (slide manipulation)is assumed to be easier than the manipulation to tilt the imaging device100 in a predetermined direction. Hereinafter, one example of such amanipulation method and display transition will be described.

FIGS. 11A to 13C schematically show a relationship between a displaytransition example in the input/output panel 101 and the posture of theimaging device 100 in the first embodiment of the present invention. InFIG. 11, an example in which a still image is displayed like a movingimage by performing a manipulation to tilt the imaging device 100 in avertical direction (the direction indicated by the arrow 305 about thearrow 303 (304) shown in FIGS. 1A and 1B) is shown. In the example shownin FIG. 11, an example in which a jogging person is a target subject isshown.

In FIG. 11A, a display example of an image displayed on the input/outputpanel 101 is shown. As described above, the representative image isdisplayed on the input/output panel 101 directly after the group imagedisplay mode is set. In this case, the manipulation support information451 and 452 superimposed on the representative image is displayed,similar to FIG. 6( c). For example, the manipulation support information451 and 452 may be displayed using a state in which the user is assumedto view, as a reference. In the example shown in FIG. 11A, the user isassumed to view the input/output panel 101 so that the jogging person isin a vertical direction. Accordingly, the manipulation supportinformation 451 and 452 is displayed so that a horizontal direction ofthe imaging device 100 (the direction indicated by the arrows 303 and304 shown in FIG. 1A) becomes a user's viewing direction. As shown inFIG. 11B, the manipulation support information 453 and 454 may bedisplayed so that a vertical direction of imaging device 100 (thedirection indicated by the arrows 300 and 301 shown in FIG. 1A) becomesthe user's viewing direction. For example, such a display may be changedbased on the posture change detection result from the posture detectionunit 120.

In FIG. 11C, a manipulation range of the imaging device 100 whenrespective images shown in FIG. 11D are displayed is shown. In thisexample, an example in which an image is advanced or returned byrotating the imaging device 100 about the arrow 303 (304) shown in FIGS.1A and 1B to change the posture is shown. The rotation angle range inthis case is (theta)10.

In FIG. 11D, a transition of the images displayed on the input/outputpanel 101 is shown. The respective images displayed on the input/outputpanel 101 shown in FIG. 11D are images obtained by continuouslyphotographing the jogging person using the imaging device 100. In FIG.11D, manipulation support information displayed to be superimposed oneach image displayed on the input/output panel 101 is omitted.

Thus, it is possible to provide a display form like viewing an originalanimation to the user by rapidly advancing an image group generated byhigh-speed photographing by gesture manipulation of the user.

In FIGS. 12A to 12C, an example in which an airplane taking off isdisplayed like a moving image by performing a manipulation to move theimaging device 100 in a horizontal direction (the direction indicated bythe arrows 303 and 304 shown in FIGS. 1A and 1B) is shown.

In FIG. 12A, a manipulation range of the imaging device 100 in whichrespective images shown in FIG. 12C are displayed is shown. In thisexample, an example in which an image is advanced or returned by movingthe imaging device 100 in the direction indicated by the arrows 303 and304 shown in FIGS. 1A and 1B to change the posture is shown. Forexample, a movement distance of the imaging device 100 when imageadvancing or image returning is performed on one image is d1. That is,image advancing or image returning can be sequentially performed bymoving the imaging device 100 by a distance d1 in the directionindicated by the arrows 303 and 304 shown in FIGS. 1A and 1B to changethe posture. The movement distance d1 may be set based on a relationshipbetween a movement distance (total distance) determined to be allowedfor movement by the user manipulation and the number of images to bedisplayed. For example, when the total distance is d and the number ofimages to be displayed is 10, the movement distance dl may be equal tod/10.

In FIG. 12B, a display example of the image displayed on theinput/output panel 101 is shown. As described above, a representativeimage is displayed on the input/output panel 101 directly after thegroup image display mode is set. In this case, the manipulation supportinformation 461 and 462 superimposed on the representative image isdisplayed, similar to FIG. 6( c). In this case, the manipulation supportinformation 461 and 462 is displayed using a state in which the user isassumed to view, as a reference, similar to the example shown in FIG.11. In the example shown in FIG. 12B, the user is assumed to view theinput/output panel 101 so that the airplane taking off goes in ahorizontal direction. Accordingly, the manipulation support information461 and 462 is displayed so that a vertical direction of the imagingdevice 100 (the direction indicated by the arrows 300 and 301 shown inFIG. 1A) becomes a user's viewing direction.

In FIG. 12C, a transition of an image displayed on the input/outputpanel 101 is shown. The images displayed on the input/output panel 101shown in FIG. 12C are images of the airplane taking off that arecontinuously photographed by the imaging device 100. In FIG. 12C,manipulation support information displayed to be superimposed on eachimage displayed on the input/output panel 101 is omitted.

In FIG. 13, an example in which a manipulation is performed to move theimaging device 100 in a horizontal direction (the direction indicated bythe arrows 303 and 304 shown in FIGS. 1A and 1B) so that a high schoolgirl is displayed as a moving image is shown.

In FIG. 13A, a manipulation range of the imaging device 100 when imagesshown in FIG. 13C are displayed is shown. In this example, an example inwhich image advancing or returning is performed by moving the imagingdevice 100 in the direction indicated by the arrows 303 and 304 shown inFIGS. 1A and 1B to change the posture is shown. For example, themovement distance of the imaging device 100 when image advancing orimage returning is performed on one image is assumed to be d2. That is,image advancing or image returning may be sequentially performed bymoving the imaging device 100 by a distance d2 in the directionindicated by the arrows 303 and 304 shown in FIGS. 1A and 1B to changethe posture. The movement distance d2 may be calculated, similar to theexample shown in FIG. 12A.

In FIG. 13B, a display example of an image displayed on the input/outputpanel 101 is shown. As described above, a representative image isdisplayed on the input/output panel 101 directly after the group imagedisplay mode is set. In this case, manipulation support information 471and 472 superimposed on the representative image is displayed, similarto FIG. 6( c). Even in this case, manipulation support information 471and 472 is displayed using a state in which the user is assumed to viewthe input/output panel 101, as a reference, similar to the example shownin FIG. 11. In the example shown in FIG. 13B, the user is assumed toview the input/output panel 101 in a state in which a high school girlis standing. Accordingly, the manipulation support information 471 and472 is displayed so that a horizontal direction of the imaging device100 (the direction indicated by the arrows 303 and 304 shown in FIG. 1A)becomes the user's viewing direction.

In FIG. 13C, a transition of an image displayed on the input/outputpanel 101 is shown. Images displayed on the input/output panel 101 shownin FIG. 13C are images obtained by continuously photographing the highschool girl using the imaging device 100. In FIG. 13C, manipulationsupport information displayed to be superimposed on each image displayedon the input/output panel 101 is omitted.

The movement information (the total movement range and the movementdistance upon image display switching) may be changed to be easilyviewed by the user manipulation according to the user's preference.Further, the movement distance used for determination upon image displayswitching (movement distance upon image display switching) may bechanged for each image based on a relationship between adjacent orneighboring images.

The movement information (the total movement angle and the movementdistance upon image display switching) may be stored in the imagemanagement information storage unit 210 to be associated with eachgroup, and a reproduction process in a group image display mode may beperformed using the stored movement information. In this case, themovement distances upon image display switching can be sequentiallyrecorded, for example, based on a locational relationship between thetarget subject (e.g., an airplane) and the imaging device 100 in theimaging operation. For example, when the airplane is a target subject, apredetermined location (an imaging location) separate from the side ofthe airplane by a predetermined distance is a reference, and a movementamount of the imaging device 100 horizontally moving from the imaginglocation is sequentially recorded.

Further, the movement information (the total movement range and themovement distance upon image display switching) may be obtained byperforming image processing on the image to be displayed. For example, afeature amount of each image is extracted and feature amounts betweenadjacent images are compared to calculate a relative displacementbetween the adjacent images. Based on the relative displacement, amovement amount between the adjacent images and a movement direction maybe calculated to determine the movement distance upon image displayswitching. As a method for calculating the relative displacement, forexample, a calculation method of calculating a relative displacement inwhich pixel values are correlated in an overlapped area between imagesmay be used. For example, a Lucas-Kanade method, a block matchingmethod, or the like can be used to obtain the relative displacement.Further, based on the obtained movement distance upon image displayswitching, the total movement range d may be obtained (e.g., d=sum ofthe movement distances (the movement distances upon image displayswitching)). For example, the group image reproduction unit 160 performscalculation of such movement information under control of the controlunit 140.

A manipulation method relating to the change of the posture of theimaging device 100 (a method of manipulating an image to be displayed(e.g., the manipulation method 215 shown in FIG. 4)) may be determinedby performing image processing on the image to be displayed. Forexample, as described above, the similarity of the feature amountsbetween adjacent images and the relative displacement between theadjacent images are calculated by comparing the feature amounts betweenthe adjacent images. Similarly, similarity of feature amounts betweenthe neighboring images and the relative displacement between theneighboring images are calculated by comparing the feature amountsbetween the neighboring images. That is, the correlativity between theimages is detected by comparing the adjacent or neighboring images.Based on such similarity and relative displacement, the manipulationmethod relating to the change of the posture of the imaging device 100(the method of manipulating an image to be displayed) may be determinedFor example, when a target subject included in an image to be displayedis moving in the image, the movement direction according to “horizontalmovement” or “vertical movement” may be determined. Further, forexample, when the similarity of the feature amount between images to bedisplayed is relatively high, the type of a subject included in eachimage (e.g., a vertical direction) according to “horizontal rotation” or“vertical rotation” may be determined. For example, the group imagereproduction unit 160 may perform the determination of the manipulationmethod under control of the control unit 140. That is, the group imagereproduction unit 160 is one example of the determination unit definedin the claims. Further, in the example shown in the foregoing, theexample in which a display state in the input/output panel 101 ischanged by sliding of the imaging device 100 in any of up, down, leftand right directions or rotation of the imaging device 100 about such adirection has been shown. Here, the display state in the input/outputpanel 101 may be changed by movement of the imaging device 100 inforward and backward directions (the arrow 306 or 307 shown in FIG. 1A)or the rotation of the imaging device 100 about the direction. Forexample, a zoom-in/out transition in an imaged image of the targetsubject when a zoom-in manipulation or a zoom-out manipulation isperformed using the imaging device 100 can be displayed by amanipulation method of moving the imaging device 100 back and forth.

(Operation Example of Imaging Device)

FIGS. 14 and 15 are flowcharts showing an example of a processingprocedure of an image content reproduction process in the imaging device100 according to the first embodiment of the present invention. In thisexample, an example in which manipulation support information is deletedwhen a certain time elapses after the group image display mode is set isshown.

First, a determination is made as to whether an image content displayinstruction manipulation is performed (step S901), and when the displayinstruction manipulation is not performed, monitoring is continued. Onthe other hand, when the display instruction manipulation is performed(step S901), the control unit 140 sets the representative image displaymode and the representative image reproduction unit 150 displays arepresentative image and non-grouped images on the display unit 111(step S902).

Subsequently, a determination is made as to whether the image advancingmanipulation or image returning manipulation is performed in a state inwhich the representative image display mode has been set (step S903).When the image advancing manipulation or the image returningmanipulation is performed (step S903), the representative imagereproduction unit 150 performs display switching of the image displayedon the display unit 111 (step S904). That is, image advancing or imagereturning of the image displayed on the display unit 111 is performed.

When the image advancing manipulation or image returning manipulation isnot performed (step S903), a determination is made as to whether thegroup image display mode setting manipulation is performed (step S905),and when the group image display mode setting manipulation is notperformed, the process proceeds to step S916. On the other hand, whenthe group image display mode setting manipulation is performed (stepS905), the control unit 140 sets the group image display mode. The groupimage reproduction unit 160 acquires each image content belonging to agroup corresponding to the representative image displayed on the displayunit 11 upon the setting manipulation, and the manipulation method fromthe image content storage unit 200 (step S906).

Subsequently, the group image reproduction unit 160 decodes the acquiredimage content, and renders the display image in the rendering memory 170based on the decoded image content (step S907). The group imagereproduction unit 160 then displays one of the images rendered in therendering memory 170 (the representative image) on the display unit 111(step S908).

Subsequently, a determination is made as to whether a certain time haselapsed after the group image display mode is set (step S909). When thecertain time has not elapsed after the group image display mode is set(step S909), the group image reproduction unit 160 displays manipulationsupport information on the display unit 111 according to the imagedisplayed on the display unit 111 (step S910). On the other hand, whenthe certain time has elapsed after the group image display mode is set(step S909), the group image reproduction unit 160 deletes themanipulation support information displayed on the display unit 11 (stepS911).

Subsequently, the analysis unit 130 determines whether the posture ofthe imaging device 100 is changed above a certain amount based on theposture change information output from the posture detection unit 120(step S912). When the posture of the imaging device 100 is changed abovea certain amount (step S912), the analysis unit 130 determines whetherthe change corresponds to a manipulation method (the manipulation method215 shown in FIG. 4) associated with a group to be displayed (stepS913). When the change corresponds to the manipulation method associatedwith the group to be displayed (step S913), the analysis unit 130outputs a display switching instruction (an image advancing or returninginstruction) for the image displayed on the display unit 11 to the groupimage reproduction unit 160 based on the change amount. The group imagereproduction unit 160 performs display switching (image advancing orimage returning) on the image displayed on the display unit 11 based onthe display switching instruction (step S914) and the process returns tostep S909. On the other hand, when the change does not correspond to themanipulation method associated with the group to be displayed (stepS913), the process proceeds to step S915.

When the posture of the imaging device 100 is not changed above thecertain amount (step S912), a determination is made as to whether therepresentative image display mode setting manipulation is performed(step S915), and when the setting manipulation is performed, the processreturns to step S902. On the other hand, when the representative imagedisplay mode setting manipulation is not performed (step S915), adetermination is made as to whether the image content displaytermination manipulation is performed (step S916), and when the displaytermination manipulation is performed, the image content reproductionprocess operation is terminated. When the image content displaytermination manipulation is not performed (step S916), a determinationis made as to whether the group image display mode has been set (stepS917). When the group image display mode has been set (step S917), theprocess returns to step S909, and when the group image display mode hasnot been set (that is, the representative image display mode has beenset), the process returns to step S903. Steps S905 to S914 are oneexample of a control procedure defined in the claims. Step S912 is oneexample of a detection procedure defined in the claims.

2. Second Embodiment

In the first embodiment of the present invention, the example in whichthe images belonging to the group corresponding to the representativeimage displayed upon the group image display mode setting manipulationare displayed upon such a setting has been shown. Here, the number ofimages belonging to the group to be displayed upon setting of the groupimage display mode is assumed to be great. In this case, all the imagesbelonging to the group can be sequentially displayed based on the usermanipulation. However, the user manipulation range is limited andsimilar images are assumed to exist in the same group. In the secondembodiment of the present invention, an example in which, when thenumber of images belonging to a group to be displayed exceeds areference upon setting of the group image display mode, the imagesbelonging to the group are subjected to an interleaving process and thenthe interleaved images are displayed is shown.

A functional configuration of the imaging device in the secondembodiment of the present invention is substantially the same as theexample shown in FIG. 3 except that the group image reproduction unit160 performs the interleaving process and then displays interleavedimages. Accordingly, the same units as those in the first embodiment ofthe present invention are denoted by the same reference numerals and adescription thereof will be partially omitted.

(Interleaving Example of Image to be Displayed Upon Setting of GroupImage Display Mode)

FIG. 16 schematically shows a flow of an interleaving process in thegroup image reproduction unit 160 according to the second embodiment ofthe present invention.

In FIG. 16( a), a display transition example when images to be displayedin the representative image display mode among the image contents storedin the image content storage unit 200 are sequentially displayed basedon an image advancing manipulation or an image returning manipulation bya user is shown. Image #56 is a representative image of grouped imagecontents, and images #51 and #63 are images of non-grouped imagecontents. Further, the display example is the same as that in FIG. 6( a)except that the images to be displayed are different, and accordingly, adescription thereof will be omitted herein.

In FIG. 16( b), a flow of an interleaving process performed on eachimage content belonging to the group corresponding to the representativeimage displayed upon a group image display mode setting manipulation isschematically shown. The number of image contents (#52 to #62) belongingto group #111 corresponding to the representative image (image #56) is11.

Here, the group image display mode setting manipulation (a manipulationto press manipulation support information 401) by the user is assumed tobe performed in a state in which the representative image of the groupedimage content (#56) is displayed on the input/output panel 101. In thiscase, the group image reproduction unit 160 acquires each image contentbelonging to the group corresponding to the representative imagedisplayed upon a group image display mode setting manipulation from theimage content storage unit 200. Subsequently, the group imagereproduction unit 160 determines whether the number of the acquiredimage contents exceeds a certain value. Subsequently, when the number ofthe acquired image contents does not exceed the certain value, the groupimage reproduction unit 160 renders each image in the rendering memory170 based on the acquired image content. On the other hand, when thenumber of the acquired image contents does not exceed the certain value,the group image reproduction unit 160 performs an interleaving processon the acquired image content and renders each image in the renderingmemory 170 based on the interleaved image content. That is, when thenumber of the acquired image contents exceeds the certain value, thegroup image reproduction unit 160 extracts a predetermined number ofimage contents upon a group image display mode manipulation undercontrol of the control unit 140. The extracted image contents are imagesto be displayed. In the interleaving process, for example, interleavingis performed so that the number of the acquired image contents is in arange of certain values.

Here, the certain value used when the determination is made as towhether the interleaving process is to be performed will be described.The certain value may be determined, for example, based on arelationship between a range in which the posture of the imaging device100 can be allowed to be changed by the user and a display switchingrange of one image in the range. For example, a case in which theimaging device 100 is rotated to the left and right about the arrows 300and 301 shown in FIG. 1A will be described. For example, the range inwhich the posture of the imaging device 100 can be allowed to be changedby the user (a rotation angle when the imaging device 100 is rotated tothe left and right) is 90 degrees and the display switching angle of oneimage in the range is 5 degrees. In this case, the certain value usedwhen the interleaving process is performed may be 18 (=90/5).

Further, the certain value may be determined, for example, based on arelationship between a time when the group image reproduction unit 160reads the image content from the image content storage unit 200 and awaiting time that can be allowed by the user. For example, the time whenthe group image reproduction unit 160 reads the image content from theimage content storage unit 200 (a time per one image content) is 100msec and the waiting time that can be allowed by the user is 3 sec. Inthis case, the certain value used when the interleaving process isperformed may be 30(=3/0.1).

Further, the certain value may be determined, for example, based on aviewing angle of the imaging element and the number of images that canbe developed in the rendering memory 170. The imaging element is, forexample, a charge coupled device (CCD) or a complementary metal oxidesemiconductor (CMOS).

Here, in the example shown in FIG. 16( b), a case in which the certainvalue used when the interleaving process is performed is 6 will bedescribed. For example, the number of image contents (#52 to #62)belonging to group #111 corresponding to the representative image (image#56) displayed upon a group image display mode setting manipulation is11. Accordingly, the group image reproduction unit 160 determines thatthe number (11) of image contents acquired from the image contentstorage unit 200 exceeds a certain value (6) and performs aninterleaving process on the acquired image contents. For example, asshown in FIG. 16( b), the group image reproduction unit 160 interleavesfive image contents from the image contents (#52 to #62) belonging togroup #111 at certain intervals (every other). In the interleavingprocess, for example, it is preferable that interleaving is performed sothat the representative image is included in the interleaved images. Forexample, the interleaving process may be performed based on theattribute of the image content or a degree of correlativity between theimages. For example, when a face of a person is included in the imagecontent, the interleaving process may be performed so that an imagehaving a high evaluation value for facial expression can bepreferentially left.

Based on the thus interleaved image contents (#52, #54, #56, #58, #60,and #62 (shown by a bold border in FIG. 16( b))), the group imagereproduction unit 160 renders each image in the rendering memory 170.

(Operation Example of Imaging Device)

FIG. 17 is a flowchart showing an example of a processing procedure ofan image content reproduction process in the imaging device 100according to the second embodiment of the present invention. Thisprocessing procedure is a variant of FIGS. 14 and 15, and differs fromthat shown in FIGS. 14 and 15 in that the interleaving process isperformed. Further, because the processing procedure is the same as thatshown in FIGS. 14 and 15 in other configurations, the same units asthose in FIGS. 14 and 15 are denoted by the same reference numerals anda description thereof will be omitted.

Each image belonging to a group to be displayed in the group imagedisplay mode and the manipulation method are acquired (step S906) andthen the group image reproduction unit 160 determines whether the numberof the acquired images exceeds a reference (step S921). When the numberof the acquired images exceeds the reference (step S921), the groupimage reproduction unit 160 performs the interleaving process so thatthe number of the acquired images is in a reference range (step S922).On the other hand, when the number of the acquired images does notexceed the reference (step S921), the process proceeds to step S907.

Thus, when there are a number of images to be displayed in the groupimage display mode, images displayed upon setting of the group imagedisplay mode can be easily viewed by performing the interleaving processso that the number of the images to be displayed is appropriate. Thatis, image advancing and returning can be performed in a range in whichthe user is allowed to manipulate the posture of the imaging device 100.Accordingly, the image advancing manipulation or the image returningmanipulation can be prevented from being performed in an unintentionalrange.

When there are a number of images to be displayed in the group imagedisplay mode, the interleaving process is performed so that the numberof the images to be displayed is appropriate, making it unnecessary forthe group image reproduction unit 160 to read all the images to bedisplayed. Accordingly, it is possible to reduce a reading time of theimages to be displayed and a waiting time until image reproduction inthe group image display mode is initiated.

As described above, according to the embodiments of the presentinvention, a manipulation to switch the representative image displaymode and the group image display mode can be easily performed by theuser manipulation, thus making it possible to easily display each imagepreferred by the user. For example, the group image display mode settingmanipulation can be performed by merely pressing the manipulationsupport information (e.g., the manipulation support information 401shown in FIG. 6( a)). Further, when the group image display mode is set,the representative image display mode setting manipulation can beperformed by merely touching the display surface.

When the group image display mode is set, the image group can be rapidlyadvanced or returned according to the gesture manipulation by the user,thus allowing the user to enjoy the gesture manipulation.

Further, when the group image display mode is set, manipulation supportinformation (manipulation guide) according to the gesture manipulationis displayed to be associated with the display image, thus leading tosmooth gesture manipulation by the user. Further, when the group imagedisplay mode is set, each image to be displayed is rendered in therendering memory 170, making it unnecessary to read the images to bedisplayed each time the image is advanced or returned, and smooth imageadvancing or image returning can be performed according to user gesturemanipulation. Further, when the group image display mode is set, imageadvancing or image returning according to a range that can bemanipulated by the user can be performed, thereby preventing imageadvancing or image returning in an unintentional area. Thus, it ispossible to perform image advancing and image returning manipulations ina manipulation range that the user finds comfortable.

When the representative image display mode is set, it is possible torealize high-speed image advancing or image returning of normalreproduction by reading and displaying only the representative image,thereby decreasing user displeasure (e.g., displeasure caused by thewaiting time). When the representative image display mode is set, thegesture manipulation is not performed, thus preventing malfunctioncaused by an unintentional user gesture manipulation.

Thus, according to the embodiments of the present invention, when aplurality of images associated with one another is displayed, each imagecan be displayed to be easily viewed.

While the imaging device has been described by way of example in theembodiments of the present invention, the embodiments of the presentinvention may be applied to an image processing device capable ofdisplaying image contents stored in the recording medium on the displayunit. For example, the embodiments of the present invention may beapplied to image processing devices, such as a mobile phone, anavigation system, and a portable media player with an imaging function.

The embodiment of the present invention illustrates one example forembodying the present invention, and the matters in the embodiment ofthe present invention and the specified matters of the invention in theclaims have a correspondence relationship, as described in theembodiment of the present invention. Similarly, the specified matters ofthe invention in the claims and the matters in the embodiment of thepresent invention having the same names as the specified matters have acorrespondence relationship. Here, the present invention is not limitedto the embodiments, and various variations may be made to theembodiments without departing from the spirit and scope of the presentinvention.

Further, the processing procedure described in the embodiments of thepresent invention may be regarded as a method including a series ofprocedures or as a program for causing a computer to execute a series ofprocedures or a recording medium having the program stored thereon. Therecording medium may be, for example, a compact disc (CD), a mini disc(MD), a digital versatile disc (DVD), a memory card, a Blu-ray disc(registered trademark) or a non-transitory storage medium or the like.

It should be understood by those skilled in the art that variousmodifications, combinations, sub-combinations and alterations may occurdepending on design requirements and other factors insofar as they arewithin the scope of the appended claims or the equivalents thereof.

The invention claimed is:
 1. An image processing device comprising: acontrol unit operable, in a first mode, to control display on a displayscreen of a representative image of a group of images and, in a secondmode, to control display on the display screen of an image of the groupof images corresponding to a posture of the device, wherein the controlunit is operable to switch between operation in the first mode and thesecond mode, and wherein, in the second mode, the control unit isoperable to control display on the display screen of informationindicating a manipulation method.
 2. The device of claim 1, wherein, inthe second mode, the images in the group of images are sequentiallydisplayed according to a change of the posture of the device.
 3. Thedevice of claim 2, wherein, in the second mode when a predeterminedchange in the posture of the device is determined, image advancing orimage returning is performed relative to a reference image of the groupset when operation of the control unit is switched from the first modeto the second mode.
 4. The device of claim 2, wherein the manipulationmethod shows that image advancing or image returning can be performed bya user performing a manipulation to rotate the image processing deviceso that the posture of the device changes.
 5. The device of claim 1,wherein, when operation of the control unit is switched to the secondmode from the first mode, the images in the group of images are renderedaccording to a predetermined rule.
 6. The device of claim 5, wherein,when the images in the group of images are rendered, the control unit isoperable to control display on the display screen of informationindicating rendering of the images in the group of images.
 7. The deviceof claim 1, wherein the information indicating a manipulation method isassociated with the group.
 8. The device of claim 1, wherein, in thesecond mode, the control unit is operable to change content of theinformation indicating the manipulation method based on a change in theposture of the device.
 9. The device of claim 8, wherein the content ofthe information indicating the manipulation method is changed inaccordance with the image of the group of images displayed.
 10. Thedevice of claim 1, wherein, when a predetermined image of the group isdisplayed in the second mode, the content of the information indicatingthe manipulation method indicates only one of image advancing and imagereturning can be performed.
 11. The device of claim 1, wherein, in thesecond mode, the control unit is operable to control display on thedisplay screen to delete the information indicating the manipulationmethod based on a change in the posture of the device.
 12. The device ofclaim 1, wherein, in the second mode, the control unit controls displayon the display screen to delete or change size of the informationindicating the manipulation method when a predetermined manipulation ofthe images of the group is performed.
 13. The device of claim 1,wherein, in the second mode, the images in the group of images aresequentially displayed based on manipulation of a manipulation elementof the device.
 14. The device of claim 1, wherein the control unitswitches from operation in the first mode to the second mode when asetting manipulation to set the second mode is performed when therepresentative image of the group of images is displayed on the displayscreen.
 15. The device of claim 1, wherein, when operation of thecontrol unit is switched to the second mode from the first mode, thecontrol unit controls display on the display screen to display therepresentative image of the group of images.
 16. The device of claim 15,wherein the control unit is operable to control display of informationindicating manipulation method at the representative image according topredetermined reference information for the representative image. 17.The device of claim 16, wherein, when a posture of the device is changedwhen the representative image is displayed, the control unit controlsdisplay of the information indicating manipulation method at therepresentative image according to a change in the posture of the devicerelative to a posture of the device when the representative image isdisplayed when the operation switches from the first mode to the secondmode.
 18. The device of claim 1, wherein, in the second mode, thecontrol unit is operable to display the images in the group of imagessequentially according to movement of the device by a predetermineddistance to change the posture of the device.
 19. The device of claim18, wherein the predetermined distance is based on a relationshipbetween a predetermined movement distance of the device and a number ofimages in the group.
 20. The device of claim 1, wherein the group ofimages is a multi-view image.
 21. A method of image processingcomprising: in a first mode, controlling display on a display screen ofa representative image of a group of images; in a second mode,controlling display on the display screen of an image of the group ofimages corresponding to a posture of the device; and switching betweenthe first mode and the second mode, wherein, in the second mode,controlling display on the display screen of information indicating amanipulation method.
 22. The method of claim 21, wherein, when thedisplay screen displaying the representative image is touched by a user,the operation is switched from the first mode to the second mode. 23.The method of claim 21, wherein, in the second mode, the informationindicating the manipulation method is displayed for a predeterminedperiod.
 24. The method of claim 23, wherein during the predeterminedperiod at least one image of the group of images is prepared fordisplay.
 25. The method of claim 23, wherein, after the predeterminedperiod is elapsed, the information indicating the manipulation method isdeleted from the display screen.
 26. The method of claim 21, wherein, inthe second mode when the posture of the device is changed horizontally,display on the display screen is changed from a first image of the groupof images to a second image of the group of images.
 27. The method ofclaim 21, wherein, in the second mode, display on the display screen ischanged from the information indicating a manipulation method to anotherinformation indicating a manipulation method.
 28. The method of claim27, wherein, when the another information indicating a manipulationmethod is displayed on the display screen, display on the display screenis changed from a first image of the group of images to a second imageof the group of images.
 29. A non-transitory recording medium recordedwith a computer-readable program executable by a computer for enablingthe computer to perform image processing, the program comprising thesteps of: in a first mode, controlling display on a display screen of arepresentative image of a group of images; in a second mode, controllingdisplay on the display screen of an image of the group of imagescorresponding to a posture of the device; and switching between thefirst mode and the second mode, wherein, in the second mode, controllingdisplay on the display screen of information indicating a manipulationmethod.
 30. The recording medium of claim 29, wherein, when the displayscreen displaying the representative image is touched by a user, theoperation is switched from the first mode to the second mode.
 31. Therecording medium of claim 29, wherein, in the second mode, theinformation indicating the manipulation method is displayed for apredetermined period.
 32. The recording medium of claim 31, whereinduring the predetermined period at least one image of the group ofimages is prepared for display.
 33. The recording medium of claim 31,wherein, after the predetermined period is elapsed, the informationindicating the manipulation method is deleted from the display screen.34. The recording medium of claim 29, wherein, in the second mode whenthe posture of the device is changed horizontally, display on thedisplay screen is changed from a first image of the group of images to asecond image of the group of images.
 35. The recording medium of claim29, wherein, in the second mode, display on the display screen ischanged from the information indicating a manipulation method to anotherinformation indicating a manipulation method.
 36. The recording mediumof claim 35, wherein, when the another information indicating amanipulation method is displayed on the display screen, display on thedisplay screen is changed from a first image of the group of images to asecond image of the group of images.